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Abstract 

In this paper, robust transceiver design with Tomlinson-Harashima precoding (THP) for multi- 
hop amplify-and-forward (AF) multiple-input multiple-output (MIMO) relaying systems is investigated. 
At source node, THP is adopted to mitigate the spatial intersymbol interference. However, due to its 
nonlinear nature, THP is very sensitive to channel estimation errors. In order to reduce the effects 
of channel estimation errors, a joint Bayesian robust design of THP at source, linear forwarding 
matrices at relays and linear equalizer at destination is proposed. With novel applications of elegant 
characteristics of multiplicative convexity and matrix-monotone functions, the optimal structure of the 
nonlinear transceiver is first derived. Based on the derived structure, the transceiver design problem 
reduces to a much simpler one with only scalar variables which can be efficiently solved. Finally, 
the performance advantage of the proposed robust design over non-robust design is demonstrated by 
simulation results. 
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I. Introduction 

Transceiver design for amplify-and-forward (AF) multiple-input multiple-output (MIMO) re- 
laying systems attracted a lot of attention recently, as it has a great potential to enhance the 
communication range of a simple point-to-point system, while providing spatial diversity and 
multiplexing gains. This system has been considered to be adopted in the emerging wireless 
systems, such as LTE-advanced, WINNER project, etc. Linear transceiver design for dual-hop 
AF MIMO relaying systems has been extensively studied in 0]|-[|6]|. In particular, joint design 
of relay forwarding matrix and destination equalizer minimizing mean- square-error (MSE) of 
data streams is discussed in [3]. Joint design of source precoder, relay forwarding matrix and 
destination equalizer minimizing MSE is investigated in [SJ — . The capacity maximization 
transceiver design has also been reported in JTJ , 0, O. On the other hand, linear transceiver 
design for multi-hop AF MIMO relaying systems with prefect channel state information (CSI) is 
discussed in Q. Furthermore, robust design, which takes channel estimation errors into account, 
is recently investigated in (8), 10, where the channel estimation uncertainty is considered as 
nuisance parameters and removed in Bayesian sense. 

For multiple-antenna systems, it is well-known that nonlinear transceivers have much better 
performance than their linear counterparts iflOll . ifTTTl . Recently, nonlinear transceiver design for 
AF MIMO relaying systems assuming perfect CSI appears in |fT2|. There are two kinds of 
nonlinear transceiver design: decision-feedback equalization (DFE) based design and Tomlinson- 
Harashima precoding (THP) based design. In fact, there exits a duality between these two designs 
021. However, as THP is performed at transmitter, it is free of error propagation compared to 
DFE based one. THP is the transmitter counterpart of the vertical BELL-Labs Layered Space- 
Time (V-BLAST) system. THP can effectively mitigate intersymbol interference or multi-user 
interference, and is also widely used as one-dimensional dirty paper coding (DPC). Due to its 
nonlinear nature, unfortunately, THP is more sensitive to channel estimation errors than its linear 
counterpart. In the presence of channel estimation errors, the performance of THP would degrade 
severely [fT3l . Therefore, robust nonlinear transceiver design is a promising way to mitigate such 
problem. This is the motivation of the current work. 

In this paper, we consider a general multi-hop AF MIMO relaying system. The THP at the 
source, linear forwarding matrices at multiple relays and linear destination equalizer matrix are 
jointly optimized under channel estimation errors at all terminals. As many design objectives 
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of THP can be considered as a multiplicatively Schur-convex or multiplicatively Schur-concave 
function, in this work, a unified optimization problem is investigated whose objective functions 
are multiplicatively Schur-convex/concave. With novel applications of results in multiplicative 
convexity and matrix-monotone functions, the optimal diagonal structure of the transceiver 
is derived. With the obtained optimal structures, the transceiver design is then significantly 
simplified and then iterative water-filling alike solutions are adopted to solve for the remaining 
unknown variables. It is found that if the objective function is multiplicatively Schur-concave, 
the proposed nonlinear transceiver design reduces to linear transceiver design. The performance 
advantage of the proposed robust design is assessed by simulations and is shown to perform 
much better than the corresponding non-robust design. 

The following notations are used throughout this paper. Boldface lowercase letters denote 
vectors, while boldface uppercase letters denote matrices. The notation Z H denotes the Hermitian 
of the matrix Z, and Tr(Z) is the trace of the matrix Z. The symbol 1m denotes an M x M 
identity matrix. The notation Z 1 / 2 is the Hermitian square root of the positive semidefinite matrix 
Z, such that Z 1 / 2 Z 1 / 2 = Z and Z 1 / 2 is also a Hermitian matrix. The symbol E{«} represents 
the statistical expectation. For two Hermitian matrices, C >z D means that C — D is a positive 
semi-definite matrix. The (n, m) th entry of a matrix Z is denoted as [Z] n m and A(Z) represents 
the vector consisting of the eigenvalues of Z. 



A. Signal Model 

In this paper, a K-hop amplify-and-forward MIMO relaying system is investigated, in which 
there is one source, one destination and K — 1 relays, as shown in Fig. [Q The source is equipped 
with 7V"i transmit antennas. The k th relay has M k receive antennas and N k+1 transmit antennas. 
The destination is equipped with Mk receive antennas. At the source, at each time slot, there 
is a N x 1 vector a = [ai,a 2 , • ■ ■ ,a7v] T to be transmitted. Specifically, the data symbols are 
chosen from M-QAM constellation with the real and imaginary parts of a k belong to the set 



As shown by Fig. [T] at the transmitter, the data vector a is fed into the a precoding unit which 
consists of a N x N feedback matrix B and a nonlinear modulo operator MOD M («). The square 

'in this paper, only square QAM is considered. 



II. Signal Model and Problem Formulation 
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matrix B is a strictly lower triangular matrix which allows data precoding in a recursive fashion 
and the MOD M (») is defined as 

Re(x) 1 



MOD M (» =x-2VM 



lm(x) 1 
2VM + 2 



(1) 



.2VM 2 

where the symbol [^J denotes the largest integer not exceeding z. The nonlinear modulo operator 
reduces the output signals into a square region [— vM, a/M) x [— yM, a/M). 

Generally speaking, nonlinear operation is more complicated to be analyzed than linear op- 
eration. To simplify the following analysis, as shown by Fig. [H the nonlinear precoder can be 
interpreted as the following linear operation as 



fc-i 



bk = a k - 22[B] kt ibi + d k 



(2) 



i=i 



where d k = 2\fMI k and I k is a complex number whose real and imaginary components are 
both integer. While we do not need to know the exact value of d k , it has the effect of reducing 
b k into the square region [— vM, a/M) x [— y/~M , a/M). The previous equation can be written 
into a compact form as 



where b = [bi, 



>N\ 



(3) 



(ijv] T > and C is a lower triangular matrix with unit 




diagonal elements, i.e., [C]^ = for k < I and [C]^ = 1. 

After the nonlinear operation, the vector b is multiplied with a precoder matrix Pi under 
a transmit power constraint Tr(P!R b P^) < P\ where P 1 is the maximum transmit power at 
the source. As the elements of a are independent and identically distributed (i.i.d.) over the 
constellation, b can also be considered as i.i.d. [14], i.e., 

R b = 2(M - l)/3I N 4 a 2 b I N . (4) 

The received signal Xi at the first relay is formulated as 

Xl = HxPib + ni (5) 

where Hi is the channel between the source and the first relay and ni is additive Gaussian noise 
with mean zero and covariance matrix R r 
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At the first relay, the received signal xi is multiplied by a forwarding matrix P 2 and then the 
resultant signal is transmitted to the second relay. The received signal at the second relay can 
be written as 



x 2 = H 2 P 2 HiPib + H 2 P 2 n! + n 2 



(6) 



where H 2 is the MIMO channel matrix between the first and second relay, and n 2 is the additive 



Gaussian noise vector at the second hop with zero mean and covariance matrix R n2 = erf I 



Similarly, at the k th relay the received signal is 



x fc = H fc P fc Xfc_ 



T 2 1 M 2 - 



(7) 



with H k and n k are the channel and additive noise at the k th hop, respectively. The covariance 
matrix of n k is denoted as R nfe = oflM k ■ Finally, for a A'-hop AF MIMO relaying system, the 
received signal at the destination is 



K 



k=l 



K-l 

E 

fc=i 



K 



n h < p < 



,l=k+l 



n k } + n K , 



(8) 



where n^Li^fe denotes Zk x ••■ x Zi. In order to guarantee the transmitted data s can be 
recovered at the destination, it is assumed that Nk and M k are greater than or equal to N Q. 

In practice, the channels are estimated and channel estimation errors are inevitable. 
Therefore, the channel H k can be expressed as 



Hi. = Ha 



AHi 



(9) 



where H fe is the estimated channels, and AH fc is the corresponding channel estimation errors 
whose elements are zero mean Gaussian random variables. Furthermore, the M k x N k matrix 
AH fc can be decomposed using the widely used Kronecker model [H, lIU, lfT51 as AH fc = 



1/2 1/2 

S fc Hw ; feW fc , where the elements of the M k x N k matrix H.w,k ^ i.i.d. Gaussian random 



variables with zero mean and unit variance. The specific formulas of ~S k and *& k are determined 
by the training sequences and channel estimators flU, EH- 



B. Problem Formulation 

As shown by Fig. Q3 at the destination, a linear equalizer G is adopted and is followed by 
a modulo operator. Notice that the effect of d will be perfectly removed by modulo operator 
at the destination and estimating a is equivalent to estimating s 0TJ. Thus at the destination, 
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a linear equalizer G is used to detect the data vector s. The MSE matrix of the data vector is 
defined as E{(Gy — s)(Gy — s) H } ifTTI . |fl4ll . where the expectation is taken with respect to 
random data, channel estimation errors, and noise. Following a similar derivation to that in $8§, 
it can be shown that 

<&(G, P fc , C) = E{(Gy - Cb)(Gy - Cb) H } 

= GfH^R^P^H^ + Tr(P i ,R XA ,„ 1 P|*„)S x + R njf ]G H 

- alG n (H fc P fc ) C H - al 

k=l 

where matrices R Xfc is defined as 

R Xfe 4 E{x fc x*} = H.PfcR^Ppl" + Tr^R^Pf *,)£, + R nfc . (11) 
It is obvious that R Xfe is the covariance matrix of the received signal at the relay. Notice that 

Rx = Rb = of lyv- 

For MEMO transceiver design, a wide range of objective functions can be expressed as a func- 
tion of the diagonal elements of the MSE matrix. For example, for sum MSE minimization, the 
objective function is /([MSEi, • ■ ■ , MSE 7V ] T )= £^ =1 MSE n , where MSE n = [$(G, P fc , C)] n>n . 
For product MSE minimization, the objective function is /([MSEi, • • • , MSEjv] t ) =IIn=i M SE„. 
Furthermore, worst-case MSE minimization corresponds to minimizing the objective function 
given as /([MSEi, • • ■ ,MSEjv] T ) = max n= i iV .,.y{MSE n } [0, (23, El, 0/7]]. On the other 
hand, weighted geometric mean MSE minimization corresponds to minimizing the following 
objective function /([MSEi, • • • , MSE 7V ] T ) = nJLi MSE^ 1 with < v 1 ■ • • < v N . Therefore, a 
unified transceiver design optimization problem can be formulated as 

min /([MSEi, • • • , MSEyv] T ) 

s.t. MSE n = [*(G,P fc ,C)] n , n 

It(P fc R Xfc _ 1 pH) < P k , k = l,---,K (12) 

where the matrix C is a lower triangular matrix with unit diagonal elements. 

In general, the objective function /(•) possesses two important properties: 
(1) /(•) is an increasing real-valued matrix function C N R, i.e., for two vectors u = 
[«i, u 2 , ■ ■ ■ , u N ] T and v = [vi,v 2 , ■ ■ ■ , v N ] T , when u n > v n , we have /(u) > /(v). This property 



K 



Gj](H fc P fc ) C H 



a b 2 CC H 



(10) 



k=l 
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is natural in transceiver design. This is because for two designs resulting in [MSEi, • • • , MSEjv] t 
and [MSEx, • • ■ , MSE^] 1 ", suppose MSE n < MSE n for all n, we will prefer the former design. 
This fact is reflected in /(•) being an increasing function. 

(2) /(•) is multiplicatively Schur-convex or concave, with definitions given below. 
Definition 1: For any z £ W l , let zw denotes the k th largest elements of z and zqa denotes the 
k th smallest elements of z, i.e., zm > • ■ ■ > Znyi and zm < ■ ■ ■ < z^ N y For two vectors v, u 
whose elements are nonnegative, v -< x u is defined as 

k k N N 

I%] < Jl^M' fc = V--,^-l and Jv M = JJup] (13) 

i=l i=l i=l i=l 

Definition 2: A function </>(•) is multiplicatively Schur-convex if and only if v u implies 
0(v) < 0(u). Notice that </>(•) is multiplicatively Schur-convex if and only if — 0(») is multi- 
plicatively Schur-concave. 

Notice that Definition 2 cannot be directly used to prove whether a function is multiplicatively 
Schur-convex or Schur-concave. In practice, we need the following Lemma 1. 
Lemma 1: Let </>(•) be a continuous real-valued function defined on V = {z : z 1 > ■ ■ ■ > z N > 
0}. Then </>(•) is multiplicatively Schur-convex if and only if for all z £ V, 

<fi( z i, • " " , Zk-i, Zkf e, z k+ i x e, Zfc + 2, • ■ • , z N ) 

is decreasing in e over the following regions 

1 < e and 2 fc /e > x e for = 1, • • • , N — 1. (14) 

Proof: See Appendix [Al ■ 

With Lemma 1 and straightforward computation, it can be proved that the four objective 
functions mentioned above are multiplicatively Schur-convex or concave. In the following, 
for notational convenience, multiplicatively Schur-convex/concave is referred to as M-Schur- 
convex/concave. 

Remark: Notice that in [TlOl . [[TT|. there is another way to prove whether a function is M-Schur- 
convex/concave. However, the method in ifTOll . [fTTTl requires all input variables z\, z 2 , z N > 0. 
In contrast, Lemma 1 provides a stronger result and allows elements of z being zero. 
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The linear minimum mean-square-error (LMMSE) equalizer is obtained by setting the differ- 
entiation of the trace of (flOl) with respect to G* to be zero, and we have 

n (H.p.) c h 



Glmmse — of 



,k=l 



+ Tr(P x R XK _ 1 P|* x )S /< + R^]- 1 . (15) 
In terms of MSE, LMMSE estimator is a dominated estimator in linear estimators |fT8il , i.e., 



$(G LM MSE,P fc ,C)^$(G,P fe ,C) 



(16) 



which implies [$(G L mmse, Pjt, C)] n<n < [*(G, P fc , C)] n<n . As /(•) is an increasing function, 
and there is no constraint on G in (fl"2l) . the optimal linear equalizer is LMMSE equalizer, i.e., 



G op t — Glmmse- 

Substituting the optimal equalizer (fl"5l) into the MSE formulation (flOl) . the MSE matrix is 
rewritten as 

K ~| 

n (a fe p fc ) c h 



*MSE(P fc ,C) = a b 2 C(l w -a 6 2 



k=l 



[H^-P^R,,,,, ,P^H H 



+ TliPKR^Pf^K^K + R 

based on which the optimization problem (fT2l) becomes 



min /([MSEi,--- ,MSEjv] 



K r K J^x k—i K K 

n (a fc p fc ) c h 

fc=i 



(17) 



s.t. MSE n = [$MSE(Pfc,C)] n ,„ 
Tr^R^P") < P k . 



(18) 



From the definition of R Xk in (fTTI) . it is noticed that R Xfe is a function of P; with / < k. In 
other words, the constraints in (fl"8l are coupled with each other. In order to simplify the analysis, 
we define the following new variables 



F a = PiRj, /2 Qo 

-1/2 



and F k = P^^CKi^Hfc^Ffc-iFjgLiH^K-^ + WJ^Qti 



H fjH 



-1/2 



,l/2 n H 



(19) 
(20) 
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where K Ffc is defined as 

K Ffc 4 Tr(F fe F** fe )£ fe + <I Mfe , (21) 



and the matrix is an additional unknown unitary matrix. Based on the definition of in <|T9l ) 
and (1201) , it is easy to show that F fc F^ = P fc R Xfc and thus the power constraints becomes 
independent of each other 

THPfeR^pH) = Tr(F fe Ff ) < P k . (22) 
Meanwhile, using (fl9l ) and (l20l) . the MSE matrix is further reformulated as 

K 

*MS E (Q*,F fc ,C) = c(l N - Q?{n[Q fc (K F y 2 H fc F fc FfH^K F y 2 + I M J" 1 / 2 K F ; /2 H fc F fc ]} H 



k=l 

k=l 



{f[lQ k (K F ; /2 H fc F fc FfH«K F ; /2 + lMj- 1/2 K Ffc 1/2 H fe F fe ]}Q )c H a 1 



a 2 b C(l N - QH MHqH M h q h . . . MlQlQ K MK • • ■ Q2M2Q1M1 Qo)C H . 

(23) 



^0 



Based on (1221) and ( 1231 , the optimization problem ( 1T8T ) is simplified as 

min /([MSEi, • ■ ■ , MSEat] t ) 

s.t. MSE„ = a\ [0(1* - Q*0Qo)C H ] nn 

= Mf Qf • • M^Q^QxM* • • ■ C^Mi 
Tr(F fc F«) < P fc , Qf Qfc = lAf fc _!- (24) 

Notice that the largest singular value of is smaller than one. Therefore, the largest 
eigenvalue of is smaller than one |fT9l and then I N — Q^0Q o is a positive definite matrix. 
With the Cholesky factorization 

(1 N - Q*0QoK 2 = LL H (25) 
where L is a lower triangular matrix, and the definition of MSE n in the second line of (124)) . we 
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have 

MSE n , = a 2 b [C(I N - Q?0Q o )C H ] n , n 
= ([C H ] : ,„) H LL H [C H ] ; , n 

n-l 
i=l 

> [L]n,n, (26) 

where D is a diagonal matrix defined as 

D = diag{[Li jl> ... ,W] T }. (27) 

In order to make the equality in the final line of (T2~6l) to hold, we need J27=i Wfil [(CLD" 1 ) 11 ]^) 2 
= 0, whose solution is 

C opt = DL- 1 . (28) 

As a result MSE n = [L] 2 n , and the optimization problem for robust transceiver design is 
formulated as 

min /([[L]? )1) --.,[L]^ Ar ] T ) 

* kMk 

s.t. ^(Ijv - Q^Qo) = LL H 

= MfQH . . . MlQlQ^Mx • • • QiMi 

Tr(F fc F«) < P fc , QfQ fc = I Mfc . (29) 

IV. Optimization Problem Reformulation for F fe 
A. Optimal Solution of Qo 

Because the objective function of the optimization problem (|29l) is M-Schur-convex or M- 
Schur-concave. In the following, we will discuss the two cases separately. 
M-Schur-convex: 

Taking the determinant on both sides of (1251) . we have 

N N 

\a 2 b (I N - Q»0Q o )| = U[L]l >n = al N - A n (0)) (30) 

n=l n=l 
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where A n (0) is the n th largest eigenvalue of 0. Based on (|30l ), the following multiplicative 
majorization relationship can be established lfT9ll 



err, 



N 



I(l-An(0)) 



n=l 



ijv -< x [[l]?,i, • • • , m 2 N,N\ > 



(3D 



where the symbol tg) denotes the Kronecker product and ljy is a x 1 all-one vector. With 
Definition 2 and /(•) being a M-Schur-convex function, (l3TT) leads to 



/([[L]? il; --- jL]^f)>/ a 



iV 



I(l-A n (0)) 



n=l 



L/V 



(32) 



%[A(©)] 



where A(0) = [Ai(0), ■ ■ • , Aat(0)] t . The equality in (|32l ) holds when -< x in (|3TT) is replaced 
by equality, which means that [L]^ n are identical for all n. Notice that from (1231) . we can write 
LL H = of Qq (I — 0)Qo- Since I — is positive definite, there always exists an unitary matrix 
Q which makes the Chrolesky factorization matrix of Qq (I — 0)Qo have identical diagonal 
elements ifTTTl . An explicit algorithm for constructing such Q is given in Appendix |B] 
M-Schur-concave: 



(33) 



(34) 



From definition of L in (1251) and based Weyl' theorem 112011 . we have 

[[L]? >1 ,---,[L]^ Ar ] T ^ x( 7 6 2 [l J v-A(0)]. 

Applying /(•) on both sides of (1331) and with Definition 2, we have 

/([[L]? jl ,---,[L]^ Ar ] T )>/K 2 [l^-A(0)]). 

v v ' 

%[A(0)] 

The equality in (l34l) holds when -<; x in (1331) is replaced by equality, which means that [L] 2 „ 
equals to of [1 — A„(0)]. On the other hand, taking eigenvalues on both sides of (1231) . we 
can obtain a 2 b [l N - A(0)] = [Aat(LL h ), • • • , Ai(LL H )] T . Therefore, [[L]^, • • • , [Lj^p = 
[Aat(LL h ), • ■ ■ ,Ai(LL H )] T , which implies L is a diagonal matrix. With L being a diagonal 
matrix, Qq 0Qo is also a diagonal matrix. This can be satisfied if we take Q = U©, where 
the unitary matrix U© is defined based on the eigendecomposition = U©A©U@ with the 
elements of A© arranged in decreasing order. 

Notice that since L is a diagonal matrix, C op t in (1281) is also a diagonal matrix. Based on the 
definition of C in © and with the fact that C is a lower triangular matrix with unit diagonal 
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elements, it can be seen that the feedback matrix B must be an all-zero matrix. Therefore, when 

the objective function is M-Schur-concave, THP becomes linear precoding. 

Summary: 

Summarizing the previous results, when the objective function is M-Schur-convex or M-Schur- 
concave, the optimization problem (|29l) is equivalent to 

min g[X(&)] 

s.t. e = M?Qi ■ ■ ■ M|Q|Q K M K • • ■ QiMi 

Tr(F fc F*) < P k , QHQ fc = I Mfc . (35) 

where g[A(0)] equals to 

rw ™ f/K[nn=i(l-An(e))]^®l w ) if /(.) is M-Schur-convex 
g[\{@)\ = < . (36) 

I f(a%[l N - A(0)]) if /(•) is M-Schur-concave 

It is difficult to directly solve the optimization problem (1351) . because is a product consists 
of matrices M^'s which in turn are complicated functions of the variables F^'s. In order to 
simplify the optimization problem (1351) . multiplicative majorization theory will be exploited, 
which transforms the objective function of (|35T) to be a direct function of F&. Below several 
useful results are given first, which form the theoretical basis of the following derivation. 

B. Prerequisites of Multiplicative Majorization Theory 

Definition 3: For two vectors v, u e V with V — {z : z\ > ■ ■ ■ > z N > 0}, v -< u u is defined 
as 

k k 

nv[i]<II«M, k = l,--- ,N. (37) 

i=l i=l 

Notice that there is a subtle difference between Definition 2 in (fT3l and Definition 3. In 
Definition 3, when k = N, Yli=i v [t] ^ Y\!i=\ u [i] rather than Ylf=i v [i\ = YliLi u [i] m Definition 
2. 

Lemma 2: Let </>(•) be a real-valued function on V. Then </>(•) is decreasing and multiplicatively 
Schur-concave on V if and only if 

v ^ n u => 0(v) > 0(u). (38) 

Proof: See Appendix O ■ 
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Lemma 3: When </>(•) is increasing and multiplicatively Schur-concave, for v, u E C = {z : 

1 > zx > ■ - ■ > z N > 0} 

v^u^^(l w -v)>^(l w -u). (39) 

Proof: See Appendix [D]. ■ 

C. Problem Reformulation 

Based on the given results of multiplicative majorization theory, the optimization problem (1351) 
can be transformed into a much simpler one. Before presenting the result, two useful properties 
of the objective function #(•) are first derived based on the multiplicative majorization theory. 
Property 1: The vector A(©) has the following relationship 

A(0) [7i({F fc }f =1 ), 7 2({F fc }f rl ),---, 7jV ({F fc }f =1 )]^ 

%({F fe }f =1 ) 

with lJ{^k\k-l) = TT r H -/ fc 1 - r , (40) 

where the equality holds when 

Q fe = V Mfc+1 U^, fc = 1, ■ ■- (41) 

where Um s and V Mfc are defined based on the singular value decomposition M fc = UM t A Mfe V^ 
with the diagonal elements of A Mfc arranged in decreasing order. Notice that (I4TT) does not cover 
Qk, but it can be any unitary matrix because it always appears in the form Q^ Qk and equals 
to an identity matrix in the objective function. 
Proof: See Appendix E ■ 

Property 2: The objective function g[A(0)] in (1351) is a decreasing M-Schur-concave function 
with respective to A(0). 

Proof: Based on Lemma 2, it is obvious that g[A(0)] is a decreasing M-Schur-concave function 
if and only if A(0) -< n A(0) #[A(0)] > g[X(@)]. In the following, we will prove the latter. 

When /(•) is M-Schur-convex, g[X(@)} = /(^[Q^iC 1 - ® 1 N )- Using Lemma 

1, rin=i(^ ~~ A n (0)) can be proved to be a M-Schur-concave function of A(0). Furthermore, 
it can be easily seen that n^=i(l ~~ A n (0)) is a decreasing function. If A(0) -<n A(0) is true, 
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based on Lemma 2, we have 

N N 
n=l n=l 

Together with the fact that /(•) is an increasing function, it is concluded that 

N N 

/K 2 [II( 1 - X n(e))} 1/N ® i) > /(^ 2 [II( 1 " A «(©))] 1/7V ® i) • («) 



n—l n=l 
.. ' 



9[A(0)] S[A(S)] 

On the other hand, when /(•) is increasing and M-Schur-concave, g[A(0)] = /(of [1^ — 
A(0)]). Using Lemma 3 we directly have A(@) -< n A(0) implies 



f(o*[l N - A(0)]) > /(^[l* - A(0)]) . (44) 



sW®)] 9[A(0)] 



Based on Properties 1 and 2, the objective function of (1351) has an achievable lower bound 
g[A(0)] > g[j({Fk}k=i)} w i m equality achieved when (|4T|) is satisfied. When the lower bound 
is achieved, we have the following three additional observations: 

(a) The constraints Q fc = l Mk are automatically satisfied. 

(b) The objective function g[j({F k }^ =1 )] is independent of Q k . 

(c) When F fc 's are known, Q fc 's can be directly computed using (14~TT) . 

Applying these three observations into (1351) . we have the reformulated optimization problem 



min <7[ 7 ({F fe }f =1 )] 

* k 



S.t. 7n({Ffc}f =1 ) - [[ — 

k=l 1 + k H k K F fc M fc* fcj 

Tr(F fc F*) < P k . (45) 
V. Solution of F k 

In the following, we first derive the optimal structure of F fc and then present an algorithm to 
solve for the remaining unknown variables. 



September 23, 2011 



DRAFT 



15 



A. Optimal Structure of F k 

Notice that g(») is a decreasing function, and "y n ({F k } k=1 ) is an increasing function of 
A n (F^H^Kp^H fc F fe ). Therefore, (^({F^}^)] is a decreasing matrix-monotone function of 
F^H^Kp^H fe F fc [|22l. Following the derivation in fl2TJ|, it can be proved that at the optimal 
solution, the power constraints hold at the equality, i.e., Tr(FfcF^) = P k , meaning that the 
relays transmit at the maximum power. 

Defining a variable r]f k as 

r) fk = a k Tr(F k F*V k ) + a 2 nh with a k = Tr(S fe )/M fc , (46) 

Tr(F fc FH) = P k is equivalent to Tr[F k Ff{a k P k ^ k + <Tv fc )]A?/ fc = P k EH, G2. Thus the 
robust transceiver design problem (1451) is equivalent to 

K 



mm 9 h({F k }f =1 )\ 



s.t. 7 n({F fc }f =1 ) = n - 



K(PfB. k i KJH. k F k ) 



k= ~ x -r A n (F«HHK F ;H fc F fc ) 



Tr[F fc F^(a fc P fc * fc + a 2 n I Nk )]/ Vfk = P k . (47) 

It is proved in Appendix |F] that when Sfr k oc I or H k oc I, the optimal solutions of the optimization 
problem (l47l) have the following structure 



F Mpt = v /^(A^)(a fe P fe * fc + <I)- 1/2 V KfejA rA^U^ bfc); 
with &(A^J = </{l - « fc Tr[V^. i7V (a fc P fc * fc + <I)" 1/2 

x * k (a k P k * k + <I)- 1/2 V Wfc)JV A|, h ]}, (48) 

where A^ fe is a A r x iV unknown diagonal matrix, and ^V<u k ,N and UArb fe ,Ar are the matrices 
consisting of the first iV columns of V^ fc and UA r b fc , respectively. The unitary matrix UA r b fe is 
an arbitrary M k _i x M fe _x unitary matrix, and the unitary matrix V^ fc is defined based on the 
following singular value decomposition 

(Kp^J-V^KPA + <I)" 1/2 = U nk A Hk Vl k (49) 
where the diagonal elements of A^ fc are arranged in decreasing order. 
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B. Computation of Kj: k 

It is obvious that in (l48l) . the only unknown variable is Ajr fc . In the following, we will discuss 
how to solve A^r fc in more detail. Denoting the following diagonal elements as 

[A"H fc ] n ,ri = hk t n, [A;F fc ]n,n = fk,n (50) 



substituting (1481) into the optimization problem (1471) and making use of ( 1491) , after a tedious but 
straightforward derivation, the optimization for robust transceiver design is simplified as 



min <?[ 7 ({F fc }f =1 )] 

J k.n 



K f2 h 2 



P h 2 

/r-n 1 K \ TT J k.n k.n 
S.t. 7n({Ffc}^ =1 ) = [[ 



-L P h 2 + 1 

=1 Jk,n lb k,n ' L 



XX = *V (51) 

71=1 

The solution of (ISTT) depends on whether /(•) is M-Schur-convex or M-schur-concave. 

1) M-Schur-convex functions: Notice that when /([MSEi,--- , MSEat] t ) is an M-Schur- 
convex function, regardless of the specific expression of /(•), the optimization problem (15TT) is 
equivalent to minimize rin=i(l — 7n({Ffe}fc=i)) iTTTj . Therefore, the transceiver design problem 
(I5TT) equals to 

N / f-fK 



Ylk=l fk.n^k,n 

mm > log 1 t? ■ ■ 

**< h V nf =1 (/i>L+i; 



n=l 

iY 



B-t. E^ = Pfc - < 52 > 



71=1 



In order to solve the optimization problem (1521) . iterative water-filling can be used to solve for 
f kyi with convergence guaranteed. More specifically, when fi/s are fixed with / ^ k, f k>i is 
computed as 

1 f-a k , n + Jal +4(1 - a k}n )a k}n h 2 k Jfi k \ 
fl ^W\ 2(1-0,,,) l ) »^"> N 

with a k , n = J] fl n hlJ{flnKn + 1) (53) 
l^k 

where fi k is the Lagrange multiplier which makes J2n=ifkn = Pk hold [|23l . Notice that this 
iterative water-filling algorithm is guaranteed to converge, as discussed in ||24~1|. 
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2) M-Schur-concave functions: When /([MSEi, • • • , MSE A r] T ) is a M-Schur-concave func 
tions, there is no unified solution. In this case, A^ k should be solved case by case. In the follow 
ing, we use the example /([MSE 1; • ■ ■ , MSE^H = Un=i MSE^ 1 for < v x < v 2 - ■ ■ v N to illus 
trate how to compute Aj? h . For this objective function, using (|36l) it follows that g[j({F k }^ =1 )] = 
a 2 ^ v% n^Ii (l - li({Fk}£ =1 )) V * and the optimization (ED is equivalent to 

N / T-fK 

min v n \og j 1 



llfc=l Jk,n' L k,r 



fk.i 

n=l 

N 



nf=i(/i>L+ 1 ). 

s.t. Y,fln = Pk- (54) 

n=l 

Equation (|54|) has the same form as (|52|) . Therefore, the solution can also be obtained by 
iterative water-filling solution. Notice that although the problem becomes linear transceiver design 
problem when /(•) is M-Schur concave, the current framework provides a generalization of linear 
transceiver design for multi-hop system. 

VI. Simulation Results and Discussions 

In this section, the performance of the proposed algorithms is assessed by simulations. In the 
following, we consider an AF MIMO relaying system where the source, relays and destination 
are all equipped with four antennas, i.e., N k = M k = 4. The estimation error correlation matrices 
are chosen as the popular exponential model [& k ] = o\pl and [S k ] = pr~^ where p t and 
p r are the correlation coefficients, and a\ denotes the estimation error variance. The estimated 
channels H fc 's are randomly generated based on the following complex Gaussian distributions 



2 \ 



H fc ~ CAf Mk ,N k (°M k ,N k , (1 ° e ' ^ k ® (55) 

such that channel realizations H fc = H fc + AH fe have unit variance. We define the signal-to- 
noise ratio (SNR) for the k th link as P k ja\ k . At the source node, four independent data streams 
are transmitted and in each data stream, = 10000 independent 16-QAM symbols are 

transmitted. Each point in the following figures is an average of 10000 trials. 

First, we compare the performance of transceiver design with M-Schur-convex versus M- 
Schur-concave objective functions. As discussed before, M-Schur-concave objective function 
corresponds to linear transceiver design. Therefore, the comparison is in fact between nonlinear 
and linear transceiver designs. Fig. [2] shows the bit error rates (BER) of the optimal transceivers 
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with sum MSE (a M-Schur-convex) objective function, and weighted geometric mean MSE (a 
M-Schur-concave) objective function. Both two-hop and three-hop systems are simulated with 
Pt = 0, p r = 0.2, a 2 = 0.001, P 2 /a 2 n2 = P z /a 2 n3 = 30dB and Pja^ being varied from 
5 to 30dB. As expected, the designs with M-Schur-convex objective function (i.e., nonlinear 
transceiver) perform much better than that with M-Schur-concave objective function (i.e., linear 
transceiver). Furthermore, although the three-hop system performs not as good as the two-hop 
system, due to the extra hop of channel and noise amplification, the performance of the two-hop 
and three-hop systems shows the same trend. In the following, we focus on the M-Schur-convex 
objective function (i.e., nonlinear transceiver) for two-hop system only. 

Next, we investigate the effect of the channel estimation error on the BER performance. Fig. [3] 
shows the BERs of the proposed robust nonlinear design and the corresponding algorithm based 
on estimated CSI only (which takes the channel estimates as true channels) with p t = 0.5, 
p t = 0, P±/a 2 = 30dB, and P^ja 2 ^ being varied from 10 to 35dB. It can be seen that smaller 
estimation errors lead to better performance for both algorithms, but the performance of the 
proposed algorithm is always better than that based on the estimated CSI only. Of course, the 
performance of the two algorithms coincide when a 2 = 0. 

Finally, we illustrate the effects of correlation in the channel estimation errors. Fig. @] shows 
the BERs of the proposed robust design with M-Schur-convex objective functions and the 
corresponding algorithm based estimated CSI only for different p r , when p t = 0, a 2 = 0.002, 
P±l a ni = 30dB, and iV^ 2 being varied from 10 to 35dB. It can be seen that in addition to the 
fact that the performance of the proposed robust design is always better than that based on the 
estimated CSI only, as p r increases, the performance gain of the proposed robust design with 
respect to that based on CSI only becomes lager. It is most obvious when p r = 0.9 and at high 
SNR at the second hop. Fig. [5] shows the corresponding BERs for different p t , with p r = 0, 
0-2 = 0.002, P 2 /o-n 2 = 30dB, and Pi/u^ being varied from 10 to 35dB. It can be seen that a 
similar conclusion can be drawn. 

VII. Conclusions 

Joint Bayesian robust transceiver design for multi-hop AF MIMO relaying systems was 
investigated. It was assumed that channel estimation errors exist in CSI in all hops. At the 
source node, a nonlinear Tomlinson-Harashima precoding was used, and was jointly optimized 
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with linear forwarding matrices at all relays and linear equalizer at the destination. A general 
transceiver optimization problem was formulated with objective function being either M-Scur- 
convex or M-Scur-concave. Using elegant properties of multiplicative majorization theory and 
matrix-monotone functions, the optimal structure of the transceivers was first derived. Then, 
the original optimization problem was greatly simplified and an iterative water-filling solution 
was proposed to solve for the remaining unknown variables. Simulation results showed that the 
proposed robust design has much better performance than the non-robust design. 

Appendix A 
Proof of Lemma 1 

Based on Definition 2, 0(z) is M-Schur-convex over V = {z : z\ > ■ ■ ■ > z N > 0} if and 
only if for v, u G V, v -< x u implies 0(v) < 0(u). 
For a vector zeD, define 

k 

z = [*!,■■• ,z N ] T and 4 = JJ^- (56) 

i=i 

For v, u G V, it is obvious that v ^; x u is equivalent to 

{vk < ^fc}f=i\ and v N = u N . (57) 
On the other hand, based on (l56l) . z k equals to 

Zk = Zk/zk-i, k < L z , (58) 

where L z — 1 is the number of the nonzero elements of z. Therefore 0(v) < 0(u) can be written 

as 

(piv^^/vi- ■ ■ ,v Lv /v Lv -i,0 r -- ,) < 0(mi,m 2 /mi • • • ,u Lu /u Lu -uO, ■ • ■ ,), (59) 
v v ' 11 v ' 

=V>(v) 

Based on (1571) and (|59l) , proving 0(z) is M-Schur-convex is equivalent to proving when {v k < 
Uk}k=i an d vn — un hold, we have ^(v) < ^(u). In other words, the proof becomes to prove 
^(•) is a vector-valued increasing function. 

To prove is increasing, we only need to prove that when v k < u k and vi = ui for all 
Ij^k, we have ip(v) < ip(u) tlT9ll . As v k > and u k > , v k < u k is equivalent to v k = u k /e 
with e > 1. Substituting S fc = n fc /e and vi = ui for all l^k into (l59l) and replacing u k = u k /u k _i 
for k < L u — 1, proving ^>(v) < ^(u) is equivalent to proving <j){u\, ■ ■ ■ ,u k /e,u k+ ie, ■ ■ ■ ) is 
increasing over e > 1 and u k /e > u k+ \e. 



September 23, 2011 



DRAFT 



20 



Appendix B 
Algorithm for Computing Q 

Following the sufficient conditions given in E6l . an explicit algorithm for constructing Q is 
given as follows. Without loss of generality, in this Appendix, for both singular value decompo- 
sition (SVD) and eigendecomposition, the elements of the diagonal singular value or eigenvalue 
matrix are assumed to be in decreasing order. 
Step 1: Define A based on the following eigen-decomposition 



(Ijv - MfQf ■ ■ • M^Q^Q^M^ • • • QMif'^l = UmAmUJJ,. 
Step 2: Initialize S = NxN and set 



(60) 



[S] 



i,i 



[A 



M 1,1 



I A H A| 1 / Ar 



[Am] i,i — [Am]at,-2v L ' V [Am]i,i 

Meanwhile, the orthogonal complement matrix of [S] : i is set to be 

-[S]i,i 
I 



A 



(61) 



Ml N,N 



\NA 







(62) 



Step 3: Begin recursion for k 



N — 2. Compute a (N — k) x (N — k) unitary matrix 



V( fc ) based on the following eigendecomposition 

(A[S]^ 1:fc ) H [I - A[S] : , 1:fc ([S]» : ,A H A[S] : , 1: ,r 1 [S]f 1:fc A H ](A[S]^) = V^A^(V^f- (63) 
Then update the (k + l) th column of S as 

[SW = [S]^v w y (fc) (64) 



and 



r (*) 



|A H A|W - [A^] N ^ N „ k 
,( fc )l 



Oi 



N-k-l, 



[A 



1,1 



|A H A| 1 / Ar 



r A (fc)i 



(65) 



[A w ]i,i - [A^]jv-fc,Ar-fc V (A 1 ' 

Based on the SVD S = UsA s Vg, the orthogonal complement matrix of [S] : i : jt + i is computed 
as 



[sii 



,l:fc+l 



[Us]:, 



k+2:N- 



(66) 
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Step 4: When k = N - 1, [S] :jJV = [SJ^^^V^-^y^- 1 ) and 

xr(^-l) - 



[A^]! x - | A H A| 1 / Ar / 1 A H A| 1 / Ar - [A^" 2 ) 



2,2 



(67) 



[A^- a )] 1(1 - [A ( ^ 2) ] 2 ,2 ' V [A (JV ^ 2) ]i,i - [A (iV " 2) ]2, 2 
Step 5: Finally, Q equals to Q = UmS. 

Appendix C 
Proof of Lemma 2 

Proof of "if" direction 

First, we will prove that for any two vectors v, u G V, v -<n u 0(v) > 0(u) implies 0(«) 
is a decreasing M-Schur-concave function over D. 

When v -<; n u =4> 0(v) > 0(u) holds, v -< x u =>- 0(v) > 0(u) must hold. Using Lemma 1, 
must be M-Schur-concave over V. 

Furthermore, for v, u G V with t> k < w fc and = Ui for all i^/c, we have v -< n u - Then 
v u =>- 0(v) > </>(u) implies 0(«) is a decreasing function. Therefore, when v -<n u =>- 
0( v ) > <K U )> then we have </>(•) is a decreasing M-Schur-convex function. 
Proof of "only if" direction 

On the other hand, when </>(•) is a decreasing M-Schur-concave function, we need prove that 
v -< n u 0( v ) > 0( u )- F° r an Y two vectors v, u G £> with v -< n u we can construct a vector 
r G X> with Tj = for i < N and tjv is chosen to makes YliLi T i = YliLi v i- h * s obvious that 
t n < u N . Then if v -< n u > we have v -< x r and t -<n u - 

As </>(•) is M-Schur-concave, based on Lemma 1 we directly have 0(v) > 4>(t). Furthermore, 
since the difference between r and u is only in the last element with tm < un, as is 
decreasing, we have 0(t) > 0(u). Combining the two inequalities, we have </>(v) < 0(u). 

Appendix D 
Proof of Lemma 3 

Based on Lemma 1, it can be proved that n!li(l — z i) * s an M-Schur-concave function. It is 
also obvious that rii=i(l — z i) * s a decreasing function for z G C — {z : 1 > z\ > ■ ■ • > Zn > 0}. 
Using Lemma 2, for v, u G C with v -< n u, we have 

n(L-^>II(i^i >0 ' fc = lr--,^. (68) 



September 23, 2011 



DRAFT 



22 



We construct a vector f = [f(i), • • • ,T(tv)] t with = for i < N and ?( N ) is chosen 
to makes rii=i^(0 = YliLi %) h°kl It is obvious that Tim > U(m. As the only difference 
between tu\ and Uu\ is at z = N, when 0(«) is increasing, we have 4>(f) > 0(u) where 

U = [U(i), • • • ,M(7V)] T . 

On the other hand, based on (1681) and the fact that f«) = ua) for z < N, it can be concluded 
that (a) nLi %) — nLi %) for 1 < fc < iV. Based on the definition of fjv, it can also 
be concluded that (b) rTi=i%) = YliLi %) > 0- Results (a) and (b) implies v -< x f where 
v = [v(i), ■ ■ ■ ,V(n)] T ttHl- As 0(«) M-Schur-concave, using Lemma 1, we have 0(v) > 0(f). 
Together with the conclusion in the last paragraph, we can obtain 0(v) > 0(u). Finally, with 
v = ljv — v and u = 1 ^ — u, the proof is completed. 

Appendix E 
Proof of Property 1 

First notice two facts in matrix theory: (a) for two matrices A and B with compatible dimen- 
sion Xi(AB) = Xi(BA); (b) for two positive semi-definite matrices A and B, YYi=i \{AB) < 
Q" =1 Xi(A)Xi(B), where the equality holds when A and B has the same unitary matrix in 
eigendecomposition [19]. With these two facts, we have 

n 

J] Xi(Mf Qf ■ • • MkQkQkM k ■ ■ ■ QxMO 

8=1 

n 

= J] A 4 (M^Q? • • ■ M|Q|Q^M K • • • Q 2 M 2 Q 1 M 1 Mf Q?) 

8=1 
/( 

< J] Ai(Mf • • ■ M*q%Q K M K • • ■ Q 2 M 2 )A i (M 1 Mf ) n = 1, • ■ • , N, (69) 

8=1 

where the first equality is due to fact (a) and the second inequality is based on fact (b). Repeating 
the above two processes and based on the fact that A^M^M^ ) = Aj(M^M fc ) we can obtain 
the following inequality 

n n 

n m©) < n M M £ M ^) A '( M "-i M ^i) • ■ • MMfMQ , (70) 

%«({F fc }f =1 ) 

where the equality holds when Q fc 's satisfy 

Qk = V Mfc+1 U^, fc = l,---,# (71) 
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where U Mfc and V Mfc are defined based on the following singular value decomposition M fc = 
UM fc A Mfc V^ fc with the diagonal elements of A Mfc arranged in decreasing order. Furthermore, 
based on the definition of M k in d2"3l> , 7;({F fc }f =1 ) in dVO]) equals to 

R JL ^(FfHfK-^fcFfc) 
7i({Ffc}fe=i) = 11 1 + Aj(F HHHK-H fc F fc) - (72) 

Appendix F 
Optimal Structure of F k 

Defining new variables 

Ffc = l/^fT k {a k P k ^ k + <I) 1/2 F fe 
and ^^(KF^^J-^H^afcPfc^ + a^I)- 1 ^, (73) 

the optimization problem (|47T) is reformulated as 

mm gM{F k }^ =1 )} 

F fe 

,t. 7„({F t } fcl )-n i + A4 ^H % p t) 

Tr(F t F]?) = P 4 . (74) 

When SE^. oc I or S fc oc I, K Ffc /?7j fe is constant [22] and thus H k is constant. Let F fe opt be the 
optimal solution of (|74l) . With the following singular value decompositions, 

UkFkppt = U Afe A Afc V5 fc , 'Hfc = U-^A^V^ (75) 

where the diagonal elements of A^ fc and A^ fc are arranged in decreasing order, we can construct 
a matrix F k equals to 

F fc = V^A Xfc V* fe (76) 

where Ax fe is a rectangular diagonal matrix with the same rank as A^ and l/b k A-n k Ax fc = A^ fc . 
The scalar b k is chosen to make that Tr(FfeF^) = P k holds. 

Using Lemma 12 in O, we can show that Ff V^ r H, k F k h Ff opt H k H k fk,o P t- Together with 
the formulation of 7n({F fc }f =1 ) in d74]), it can be concluded that 7„({F fc }f =1 ) > 7n({F fe , op t}*L 1 ). 
Since #(•) is an decreasing function, g[-y({F k }£ =1 )] < g[y({F k>opi }^ =1 )]. Because f\ opt is the 
optimal solution, it is impossible to have ^^({FfcjfL^] < fi^T^FVopt}^)]. Therefore, F k 
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must be the optimal solution. Furthermore, based on the relationship between of F k and F fc , it 
follows that 



Notice that in general the unitary matrix depends on the optimal solution F fc opt . However, 
from (1741) . it can be seen that the value of does not affect the objective functions and 
therefore it can be an arbitrary unitary matrix. Meanwhile, as the minimum dimension of 
F\ 1 'H k 1 'H k F k is N, only N x N principal submatrix of Ax fe can be nonzero. For notational 
convenience, we denote that [Ax fc ]i:jv,i : jv = A^ fc . 

Substitute (1771) into the definition of r}f k in (l46l) . we obtain a simple linear function of rjf x , 
and r)f x can be easily solved to be 



Vfh = </{l - a k Tr[V* ktN (a k P k * k + a 2 n ir l ^ k (a k P k <f k + ^r 1 ^^]} 
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Fig. 1. Multi-hop AF MIMO relaying system with Tomlinson-Harashima Precoding at the source. 




Fig. 2. BERs of the proposed transceivers with M-schur-convex and M-Schur-concave objective functions when pt = 0, 
p r = 0.2, a'i = 0.001. 
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Fig. 3. BERs of proposed robust design with M-schur-convex objective functions and the algorithm based on estimated CSI 
only when p t = 0.5, p r = 0, and P\/a^ ll = 30dB. 
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Fig. 4. BERs of proposed robust design with M-schur-convex objective functions and the algorithm based on estimated CSI 
only with different p r , when p t = 0, o\ = 0.002 and Pi/ff^ = 30dB. 
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Fig. 5. BERs of proposed robust design with M-schur-convex objective functions and the algorithm based on estimated CSI 
only with different p u when p r = 0, al = 0.002 and P 2 /o"n, = 30dB. 
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